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At the time IDEA student ratings of instruction were first 
developed, most college and university courses were 
taught on campus. With the growth of the Internet, 
more and more courses are being offered online. Since 
2002, the IDEA Student Ratings of Instruction system 
has been available through an online survey delivery 
system - IDEA Online. Initially, online survey 
administration was used primarily for online courses. 
However, the use of online surveys has increased each 
year in both online and traditional courses. 


This report summarizes findings from IDEA Technical 
Report No. 15 (Benton, Webster, Gross, & Pallett, 
2010), which compared IDEA ratings in traditional (i.e., 
on campus) and online courses. That study examined 
similarities and differences in students’ response rates 
and instructor and student ratings in classes identified 
exclusively as either traditional or online. The following 
sections describe the sample investigated, methods 
employed, and key findings from the analyses 
contained in Technical Report No. 15. Conclusions and 
recommendations are provided at the end of the 
report. 


The Sample 


The sample of classes came from institutions that 
administered the IDEA Student Ratings of Instruction 
system with IDEA online between 2002 and 2008. A 
total of 5,272 classes from 38 institutions were 
identified as exclusively traditional and 13,416 classes 
from 67 institutions were classified as exclusively 
online. The classes were representative of the overall 
IDEA database and of all ratings administered online. 
In addition, the principal types of students enrolled in 
the courses were, for the most part, consistent across 
modalities. The percentage of IDEA Online use in 
traditional, face-to-face courses increased from 6.5% in 
2002 to 31.9% of all use in 2008. The number of 
online courses has dramatically increased within the 
same time period from 216 classes in 2002 to 6,361 
classes in 2008. 


The IDEA Student Ratings of Instruction System 
Instructors in each course completed the Faculty 


Information Form (FIF), which is used to solicit 
information about the course. They rated 12 learning 


objectives as either 1=Essential, 2=Important, or 3=of 
Minor or No Importance; they also responded to 
additional questions about the course. 


Students completed either the IDEA Diagnostic Form or 
Short Form. Students rated their progress on each of 
the 12 learning objectives using a five-point scale of 
1=No apparent progress, 2=Slight progress, 
3=Moderate progress, 4=Substantial progress, and 
5=Exceptional progress. Students also rated other 
aspects of the course and responded to questions 
about their typical work habits, motivation to take the 
course, and effort. Students using the Diagnostic Form 
also reported how frequently their instructor employed 
each of several teaching methods, using the scale 
1=Hardly Ever, 2=Occasionally, 3=Sometimes, 
4=Frequently, and 5=Almost Always. 


Response Rates to Online Surveys in Traditional and 
Online Courses 


Response rate indicates how representative course 
ratings are of all students enrolled in the class. 
Between 2002 and 2008, among classes using IDEA 
Online, students in traditional courses had an average 
response rate of 58% compared to 50% for students 
enrolled in online courses. Notably, the overall average 
response rate using IDEA Online (55%) is lower than 
the paper-and-pencil version (78%). 


Response rates in this sample varied by class size. For 
both traditional and online courses, the highest 
response rates were found in classes enrolling fewer 
than 10 students (traditional courses, 64%; online 
courses, 58%). Online courses with enrollments 
exceeding 39 students had the lowest response rate 
(41%). 


Many explanations could be given for why response 
rates were somewhat higher in traditional courses and 
smaller classes. Perhaps because of face-to-face 
contact with students, an instructor in a traditional 
course has more influence. This is probably even more 
common in smaller classes. Some traditional 
instructors also may have access to computer labs 
where students can complete the ratings in-class. 
Others may work in institutions where students are 
encouraged to bring laptops to class. In contrast, most 


online instructors never meet students in person. The 
lack of face-to-face contact may diminish the 
instructor’s influence on student compliance. (For 
recommendations to improve IDEA Online response 
rates, see Facilitating Response Rates in IDEA Online, 


www.theideacenter.org/OnlineResponseRates). 
Instructor Ratings of 12 Learning Objectives 


A basic premise of IDEA student ratings is that types of 
learning must reflect the instructor’s purpose. This is 
why instructors rate the importance of the 12 learning 
objectives on the FIF. They identify which learning 
outcomes are most essential. Findings from Technical 
Report No. 15 indicated similarities between online 
and traditional courses in average ratings of objectives 
and the percent of instructors rating each objective as 
essential, important, or of minor/no importance. Inter- 
correlations among instructor ratings of the 12 
objectives were also very similar, which implies the 
associations instructors make between the objectives 
are consistent across course types. Expectations for 
learning outcomes were consistent between online and 
traditional courses. The 12 learning objectives are, 
therefore, relevant for both course delivery formats. 


Student Ratings of Learning Objectives and Teaching 
Methods 


One of the underlying assumptions of the IDEA 
Students Ratings of Instruction system is that 
effectiveness is determined by students’ progress on 
objectives stressed by the instructor. Students should, 
therefore, report greater progress on objectives the 
instructor designates as either essential or important. 
In both traditional and online courses, average student 
ratings were, in fact, higher for essential and important 
objectives than for minor or unimportant objectives, 
supporting the use of IDEA in either course delivery 
format. 


Students also rated how frequently the instructor used 
various teaching methods. The research investigated 
whether students reported differences in the frequency 
of 20 teaching methods utilized by the course 
instructor. The teaching methods represented five 
areas: Stimulating Student Interest; Fostering Student 
Collaboration; Establishing Rapport; Encouraging 
Student Involvement; and Structuring the Classroom 
Experiences. As documented in Technical Report No. 
15 (2010, Table 13), on the whole, students’ ratings of 
the teaching methods were highly similar across course 
type. This means students in online and traditional 
courses believed their instructors employed the 
teaching methods with similar frequency. However, 


there was one notable exception. Online students 
reported greater instructional use of educational 
technology to promote learning (Item 47). This 
difference makes sense when one considers that, by 
their very nature, online courses rely heavily upon 
technology. These results suggest that the pedagogical 
principles underlying the 20 teaching methods are 
employed when courses are taught face-to-face or in an 
online learning environment. 


The Relationship between Instructor and Student 
Ratings of Learning Objectives 


An indirect test of the validity of the IDEA system 
involves correlating students’ reported progress for 
each learning objective with the instructors’ ratings of 
the importance of those objectives (See Hoyt & Lee, 
2002a). The highest correlations should be found in 
ratings of the same objectives (i.e., students rate the 
most learning on essential objectives and the least 
learning on objectives not stressed in the course). This 
hypothesis was supported, as correlations among 
ratings of the same objectives were on average higher 
in both traditional and online courses (Benton, et al., 
2010, see Tables 15-18). Evidence for validity of the 
IDEA objectives is, therefore, found in both types of 
courses. 


The Relationship between Student Ratings of Learning 
Objectives and Teaching Methods 


In the IDEA Diagnostic Form Report, suggestions are 
made for improving teaching effectiveness based on 
correlations between students’ ratings of progress on 
the 12 learning objectives and their ratings of the 
instructor’s teaching methods. In order for the 
suggestions to be of value to instructors utilizing both 
course modalities, the pattern of correlations should be 
similar. The findings indicate that the suggestions are 
valuable, as the strength and pattern of correlations 
were very consistent across type of course (Benton, et 
al., 2010, Table 19). 


The only meaningful difference was found in the 
correlation between the instructor’s use of educational 
technology (Item 47) and students’ self-reported 
progress on relevant (important or essential) 
objectives. The correlation in online courses was higher 
(r = .50) than in traditional courses (r = .35). As one 
might expect, using technology to promote student 
learning is more strongly associated with students’ 
progress on relevant objectives in the online format. 
Although there may be additional teaching methods 
and strategies to facilitate learning in either the 
traditional or online learning environment, this 


research demonstrates support for the usefulness of 
the feedback provided by IDEA to guide faculty 
reflection about their teaching and student learning. 


The Relationship between Student/Course 
Characteristics and Global Ratings 


A unique feature of IDEA is adjusted scores, which 
control for extraneous factors that influence student 
ratings. Student characteristics (e.g., work habits, 
motivation, and course effort) and course 
characteristics (e.g., enrollment, difficulty) can affect 
instructors’ scores, but such characteristics are beyond 
the control of the instructor. Students are, therefore, 
asked to rate their typical work habits, desire to take 
the course regardless of who taught it, effort in the 
course, and perceived difficulty of the course. 
Enrollment is reported by the instructors on the FIF. 


Students’ global ratings and their progress on relevant 
objectives (PRO) are then adjusted for their correlations 
with student/course characteristics. Therefore, it is 
important to investigate the similarity of those 
correlations across types of courses. Global ratings 
include three items: “As a result of taking this course, | 
have more positive feelings toward this field of study,” 
“Overall, | rate this instructor an excellent teacher,” and 
“Overall, | rate this course as excellent.” In this study, 
the pattern of correlations between student/course 
characteristics and global ratings was very consistent 
across traditional and online courses (Benton et al., 
2010, Table 20). Adjustments to raw scores made in 
traditional courses are, therefore, generalizable to 
online courses. 


Approaches to Instruction and Teaching Styles 


IDEA Research Note #1 (2003) reported the 
importance of five teaching approaches to the three 
global ratings of teaching effectiveness. The five 
teaching approaches were (1) Stimulating Student 
Interest, (2) Fostering Student Collaboration, (3) 
Establishing Rapport, (4) Encouraging Student 
Involvement, and (5) Structuring the Classroom 
Experiences. Students’ ratings on the teaching 
approaches were more important to ratings on the 
“Excellent teacher” item compared to the other global 
ratings of teaching effectiveness. So, whether or not 
students regard the instructor as “excellent” has much 
to do with the teaching approaches she/he utilizes. To 
determine whether this finding would be consistent in 
both online and traditional courses, the analyses from 
the 2003 study were replicated for Technical Report 
No. 15. As before, the five teaching approaches were 
most important for understanding the “Excellent 


teacher” item, regardless of course type. For traditional 
courses, the five scales explained more variance in the 
“Excellent teacher” item (R2 = .83) than in the 
“Increased positive feeling toward the field” (R2 = .60) 
and “Excellent course” (R2 = .66) items. For online 
courses, the pattern was the same: “Excellent 
teacher” (R? = .82), “Increased positive feeling toward 
the field” (R2 = .52), and “Excellent course” (R?= .67). 
This suggests that high ratings on “Excellent teacher” 
have much to do with how frequently the instructor 
employs the various teaching approaches in both 
formats. 


A replication of IDEA Report #4 (Hoyt & Lee, 2002b) 
was also conducted by investigating the relative 
importance of six teaching styles for facilitating student 
progress on specific learning objectives. The six 
teaching styles originated from combinations of the five 
teaching approaches. It seemed important to 
investigate whether these teaching styles would be 
similar across course modalities. 


The results from this analysis were consistent across 
traditional and online courses for the six teaching 
styles with only three exceptions. First, Fostering 
Student Collaboration is more helpful in traditional 
courses when the focus is on values development and 
critical analysis/evaluation. Instructors can facilitate 
collaboration by asking students to help each other 
understand ideas or concepts, forming teams or 
discussion groups, and by asking students to share 
ideas with others whose background and viewpoints 
differ. Second, Structuring the Classroom Experiences 
may be somewhat more important in online courses if 
the instructor wishes to help students achieve a broad 
liberal education. Online instructors can provide 
structure by explaining the course material clearly and 
concisely, communicating expectations, following a 
course schedule, and providing timely and frequent 
feedback. Finally, when the outcome is for students to 
find and use resources, Establishing Rapport may be 
somewhat more important in online courses. Online 
instructors can build rapport by helping students to 
answer their own questions, explaining the reasons for 
criticisms of students’ work, and displaying a personal 
interest in students and their learning. 


Summary and Conclusions 


The findings reported here provide evidence of the 
validity of IDEA student ratings in both traditional and 
online classrooms. Instructor ratings of the importance 
of the 12 learning objectives and student progress on 
those objectives are highly similar. Moreover, 
regardless of course format, students make greater 


progress on relevant (important or essential) 
objectives. 


The pattern of relationships between student ratings of 
teaching methods and learning objectives is consistent 
across modalities, providing support for the diagnostic 
use of the instrument. Furthermore, the magnitude and 
direction of correlations between student/course 
characteristics and global ratings, which underlie 
adjusted scores, are generally the same, making the 
use of adjusted scores comparable in both settings. 
Finally, the teaching approaches instructors apply in 
the classroom are associated with teaching excellence 
in both course formats. 


Only a few minor differences were found. First, 
response rates are somewhat higher for traditional 
than online courses. Second, instructors in online 
courses reportedly use educational technology more 
frequently to promote student learning. Third, more 
frequent use of educational technology is an important 
teaching method related to greater student progress on 
relevant objectives in online environments. Fourth, 
Fostering Student Collaboration is more helpful in 
traditional courses when the focus is on values 
development and critical analysis/evaluation. Fifth, 
Structuring the Classroom Experiences may be 
somewhat more important in online courses if the 
instructor wishes to help students achieve a broad 
liberal education. Finally, when helping students to find 
and use resources, Establishing Rapport may be 
somewhat more important in online courses. 


While this research supports the use of IDEA in online 
learning environments, The IDEA Center recognizes 
that no single survey can anticipate the unique needs 
of every learning situation. The use of additional 
questions may be helpful in addressing areas not 
covered in the IDEA instrument, but important to a 
particular course or learning environment. Using 
Additional Questions for Online Learning Environments 
has been developed to serve as a guide to maximizing 
the feedback obtained through the IDEA Student 
Ratings of Instruction System. 
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